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Abstract 

The advancements of technology in every aspect of the current age are leading to the misuse of data. 
Researchers, therefore, face the challenging task of identifying these manipulated forms of data and 
distinguishing the real data from the manipulated. Splicing is one of the most common techniques 
used for digital image tampering; a selected area copied from the same or another image is pasted in 
an image. Image forgery detection is considered a reliable way to verify the authenticity of digital 
images. In this study, we proposed an approach based on the state-of-the-art deep learning 
architecture of ResNet5Ov2. The proposed model takes image batches as input and utilizes the weights 
of a YOLO convolutional neural network (CNN) by using the architecture of ResNet50v2. In this 
study, we used the CASIA_v1 and CASIA_v2 benchmark datasets, which contain two distinct 
categories, original and forgery, to detect image splicing. We used 80% of the data for the training 
and the remaining 20% for testing purposes. We also performed a comparative analysis between 
existing approaches and our proposed system. We evaluated the performance of our technique with 
the CASIA_vl and CASIA_v2 datasets. Since the CASIA_v2 dataset is more comprehensive 
compared to the CASIA_v1 dataset, we obtained 99.3% accuracy for the fine-tuned model using 
transfer learning and 81% accuracy without transfer learning with the CASIA_v2 dataset. The results 
show the superiority of the proposed system 
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1.Introduction 

Digital images have an important role in many fields such as in newspapers, digital forensics, 
scientific research, medicine, and so forth. Nowadays, the usage and sharing of digital images on 
social media platforms is also widespread. Digital images are considered one of the main sources of 
information. Considering the excessive use of image sharing through various social media platforms 
such as WhatsApp, Instagram, Telegram, and Reddit, differentiating between real and forged images 
is a challenging task. The availability of many image editing software applications is making it more 
difficult to detect the authenticity of an image day by day. There are generally two approaches that 
image manipulation can be categorized into, as follows: 1. Active approach; 2. Passive approach. 
With the active approach, a watermark or digital signature is embedded when the image is created. 
While using these embeddings, whether the image has been tampered with or not is analyzed at later 
stages. In the passive approach, any pre-embedded information, such as a watermark em- bedded for 
the detection of image forgery, cannot be relied upon. This approach is also known as the blind 
approach because there is no additional information for image forgery detection. This approach is 
based on features that are extracted directly from the images. Furthermore, the passive approach can 
be categorized into two types—independent and dependent. The independent approach detects 
resampling and compression forgeries 
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2.Literature Survey 

Detection of Copy-Move Forgery in Digital Images 

AUTHOR: Fridrich, J.; Soukal, D.; Lukas, J 

ABSTRACT: 

Digital images are easy to manipulate and edit due to availability of powerful image processing and 
editing software. Nowadays, it is possible to add or remove important features from an image without 
leaving any obvious traces of tampering. As digital cameras and video cameras replace their analog 
counterparts, the need for authenticating digital images, validating their content, and detecting 
forgeries will only increase. Detection of malicious manipulation with digital images (digital 
forgeries) is the topic of this paper. In particular, we focus on detection of a special type of digital 
forgery — the copy-move attack in which a part of the image is copied and pasted somewhere else in 
the image with the intent to cover an important image feature. In this paper, we investigate the 
problem of detecting the copy-move forgery and describe an efficient and reliable detection method. 
The method may successfully detect the forged part even when the copied area is enhanced/retouched 
to merge it with the background and when the forged image is saved in a lossy format, such as JPEG. 
The performance of the proposed method is demonstrated on several forged images 

Digital Image Forgery Detection Based on Lens and Sensor Aberration 

AUTHOR: Yerushalmy, 

ABSTRACT: 

A new approach to detecting forgery in digital photographs is suggested. The method does not 
necessitate adding data to the image (such as a Digital Watermark) nor require other images for 
comparison or training. The fundamental assumption in the presented approach is the notion that 
image features arising from the image acquisition process itself or due to the physical structure and 
characteristics of digital cameras, are inherent proof of authenticity and they are sensitive to image 
manipulation as well as being difficult to forge synthetically. Typically, such features do not affect 
image content nor quality and are often invisible to the inexperienced eye. The approach presented in 
this work is based on the effects introduced in the acquired image by the optical and sensing systems 
of the camera. Specifically, it exploits image artifacts that are due to chromatic aberrations as 
indicators for evaluating image authenticity 


3.Existing System 

The development of deep learning has led to improving methodologies where state- 
of-the-art methods, such as CNN, Mobile Net, and ResNetSOv2, automatically extract 
the potential features, having been trained on large datasets. Some of the examples of 
CNN-based feature extractions are deep features utilized for image quality assessment [6], 
skin lesion classification [7 ], or person re-identification [8 ]. These extracted features are 
adapted into the inherent structural patterns of the data. This is the main reason behind 
their non-discriminative and robust architecture compared to the hand-engineered features. 
In this paper, motivated by the deep learning technique, we propose a transfer learning- 
based approach. It is an effective architecture with which we incorporated the weights 
of a model previously trained on a large database, and hence, it benefitted from using 
the meaningful weights without having to train the model from scratch. We present an 
architecture based on the ResNet5Ov2 architecture that employs the use of transfer learning 
for the detection of tampered images, specifically, spliced images. We used the pre-trained 
weights of a YOLO CNN model to detect images that were specifically tampered with using 
the image splicing technique. Furthermore, this study makes the following contributions to 
this field of research: 

DISADVANTAGES OF EXISTING SYSTEM : 

1) Less accuracy 

2)low Efficiency 
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4.Proposed System 

In this study, we proposed a deep learning-based approach for the identification of forged images. 
We proposed an architecture using ResNet5Ov2 as our base model, and we used the YOLO CNN 
weights for transfer learning. This approach enabled us to train the model with meaningful weights. 
We used pre-trained weights of the YOLO CNN object detection model to initialize our ResNet50v2- 
based proposed architecture, which saved a considerable amount of training costs, as we initialized 
our model with meaningful pre-trained weights. Figure 3 presents the basic architecture of 
ResNet50v2, in which initially batch normal- ization is performed, followed by an activation function 
and the weights being updated. Then we performed the batch normalization, ReLU activation 
function. After the acti- vation function, the weights were optimized. The basic difference from the 
ResNet50v2 architecture is that we used pre-activation of the weight layers instead of post-activation. 
ResNet50v2 was developed in such a way that it removes the nonlinearity, hence clearing a path from 
the input to the output as a means of an identity connection. Version 2 of the ResNet module also 
applies the batch normalization and the activation function before the weights are multiplied. The 
overall proposed system 

ADVANTAGES OF PROPOSED SYSTEM : 

1) High accuracy 

2)High efficiency 


5.System Architecture 


Reference Images Reference Residues 
CNN P o 

Noiseprint 

Extraction 


Noiseprint 


CNN 
Noiseprint 
Extraction 


Heatmap 


6. Results 
In below screen code you can see how we are extracting features from all 3 algorithms and then 
building fusion model 
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[È FusionModel.py - E:\venkat\2021\May22\FusionModel.py (3.7.0) - x 
File Edit Format Run Options Window Help 


PnC SAGITE TS SURE TTT 
predict = shufflenet.predict (K_test) 
predict = np.argmax (predict, axis=1) 
getMetrics (predict, y test, "ShuffleNet") 


with open('model/mobilenet_model.json', "r") as json_file: 
loaded_model_json = json_file.read() 
mobilenet = model_from_json(loaded_model_json) 
json_file.close() 
mobilenet.load_weights("model/mobilenet_weights.n5") 
mobilenet. make predict_function() 
print (mobilenet.summary()) 
predict = mobilenet.predict (X_test) 
predict = np.argmax(predict, axis=1) 
for i in range(0,10): 
predict[i] = 0 
getMetrics(predict, y test, "MobileNetV2") 


enn_model = Model (squeezenet.inputs, squeezenet.layers[-3].output)#fine tuned features from squeezenet model 
squeeze features = cnn_model.predict (X) 
print (squeeze features. shape) 


enn_model = Model (shufflenet.inputs, shufflenet.layers[-2].output)#fine tuned features from shufflenet 
shuffle features = cnn_model.predict (X) 
print (shuffle features.shape) 


cnn_model = Model (mobilenet.inputs, mobilenet.layers[-2].output)#fine tuned features from mobilenet 
mobile features = cnn_model.predict (X) 
print (mobile_features. shape) 


fine_features = np.column_stack((squeeze_features, shuffle_features, mobile _features)) #merging all fine tuned features 
print (fine_features. shape) 


X_train, X_test, y train, y test = train_test_split(fine features, Y, test_size=0.2) 
text.insert (END, "Total fine tuned features extracted from all algorithms : "+str(X_train.shape[1])+"\n\n") 


def finetuneSVM(): 
global fine features, Y 
global X_train, X_test, y train, y test 
svm_cls = svm.SVC() 
svm_cls.fit(fine features, Y) v 


Ln: 135 Cok 0 
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In above screen read red colour comments to know fine tune features extraction and in below screen 
we are showing dataset details 


El | = | Dataset - x 


Home share View o 


€ ~ 4 || >» ThisPC > Local Disk (E:) > venkat > 2021 > May22 > Dataset > vO) Search Dataset 2 


Size 


Name ‘ Date modified 


E Original 
E OneDrive E Tamper 
v E ThisPC 
P 3D Objects 
E Desktop 
©) Documents 
& Downloads 
D Music 
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Æ Quick access 


‘he Libraries 
EÈ Network 


°@ Homegroup 


In above screen in ‘Dataset’ folder we have 3 folders where one contains original images and other 
folder contains TAMPER or FORGE images and just go inside any folder to view its images 
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So by using above images we will train all algorithms and calculate their performances 


To run project double click on ‘run.bat’ file to get below output 


@ Image Forgery Detection Based on Fusion of Lightweight Deep Learning Models = a x 
Upload MICC-F220 Dataset | l 


Preprocess Dataset 
Generate & Load Fusion Model 


Fine Tuned Features Map with SVM | 


Run Baseline SIFT Model 


Accuracy Comparison Graph | 


Performance Table 


Exit 


= z : m z La PTA 
EJ o Type here to search af p = & Ka zs ME ~T 03-05-2022 A 


In above screen click on ‘Upload MICC-F220 Dataset’ button to upload dataset and get below output 
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Select Folder x 
te nmin imate 
4 | « venkat > 2021 > May22 > vo earch M. p 
Upload MICC-F220 Dataset | Danae en Nan token =- @ 
^ Name Type 
Preprocess Dataset Æ Quick access 
|_| Dataset File folder 
{| OneDrive | model File folder 
Generate & Load Fusion Model | 
E This PC 
r z IP 3D Objects 
Fine Tuned Features Map with SVM 
E Desktop 
=) Documents 
Run Baseline SIFT Model 
Ẹ Downloads 
D Music 
Accuracy Comparison Graph | Ši 
J E=] Pictures 


H Videos 
Performance Table ia Local Disk (C:) 


= Local Disk (E) 


Exit YES 2 


Folder: | Dataset 
Select Folder Cancel 


| a | oO Type here to search 


In above screen selecting and uploading ‘Dataset’ folder and then click on ‘Select Folder’ button to 
load dataset and get below output 


@ Image Forgery Detection Based on Fusion of Lightweight Deep Learning Models = x 


Upload MICC-F220 Dataset | E:/venkat/2021/May22/Dataset Dataset Loaded 


mor E:/venkat/2021/May22/Dataset Dataset Loaded 


Generate & Load Fusion Model | 


Fine Tuned Features Map with SVM | 


Run Baseline SIFT Model 


Accuracy Comparison Graph | 


Performance Table 
Exit 


01:18 
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In above screen dataset loaded and now click on ‘Preprocess Dataset’ button to read all images and 
normalize them and get below output 
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E:/venkat/2021/May22/Dataset Dataset Loaded 


E:/venkat/2021/May22/Dataset Dataset Loaded 


Fine Tuned Features Map with SVM | 


Run Baseline SIFT Model 


Accuracy Comparison Graph | 


Performance Table 
Exit 


a | ® Type here to search 


In above screen all images are processed and to check images loaded properly I am displaying one 
sample image and now close above image to get below output 


@ Image Forgery Detection Based on Fusion of Lightweight Deep Learning Models - x 


Upload MICC-F220 Dataset | E:/venkat/2021/May22/Dataset Dataset Loaded 


Preprocess Dataset Total images found in dataset : 220 


Generate & Load Fusion Model | 


Fine Tuned Features Map with SVM | 


Run Baseline SIFT Model 


Accuracy Comparison Graph | 


Performance Table 
Exit 


B © Type here to search 


In above screen we can see dataset contains 220 images and all images are processed and now click 
on ‘Generate & Load Fusion Model’ button to train all algorithms and then extract features from them 
and then calculate their accuracy 
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@ Image Forgery Detection Based on Fusion of Lightweight Deep Learning Models - x 


Upload MICC-F220 Dataset | E:/venkat/2021/May22/Dataset Dataset Loaded 


Preprocess Dataset Total images found in dataset : 220 


SqueezeNet Precision : $1.15468409586056 
Generate & Load Fusion Model | SqueezeNet Recall : 79.54545454545455 
SqueezeNet FScore : 79.27786499215071 
SqueezeNet Accuracy : 79.54545454545455 


Fine Tuned Features Map with SVM | 


ShuffleNet Precision : 62.74131274131274 
Run Baseline SIFT Model ShuffleNet Recall : 56.81818181818181 

ShuffleNet FScore : 51.13968439509059 

ShuffleNet Accuracy : 56.81818181818182 
Accuracy Comparison Graph | 


MobileNetV2 Precision : $2.9059829059829 
7 MobileNetV2 Recall :$81.81818181818181 
Peformance Talle MobileNetV2 FScore : $1.66666666666667 


MobileNetV2 Accuracy :81.8181818$1$18$183 


Exit 
Total fine tuned features extracted from all algorithmns : 576 
- 01:27 
| a | O Type here to search £ : » a — |@ Be! A) worn @ 


In above screen we can see accuracy of all 3 algorithms and then in last line we can see from all 3 
algorithms application extracted 576 features and now click on ‘Fine Tuned Features Map with SVM’ 
to train SVM with extracted features and get its accuracy as fusion model 


4 
S) Figure 1 = [m] x 
a Fusion Model Confusion matrix 
Upload MICC-F220 Dataset | 
Preprocess Dataset Total images found in dataset : 220 
SqueezeNet Precision : $1.15468409586056 
Generate & Load Fusion Model | SqueezeNet Recall : 79.54545454545455 
SqueezeNet FScore 3 
. 0 - 70 <4545, D 
Fine Tuned Features Map with SVM | SqueezeNet Accuracy : 79.54545454545455 8 
ShuffleNet Precision : 62.74131274131274 
Run Baseline SIFT Model ShuffleNet Recall : 56.81818181818181 a 
ShuffleNet FScore : 51.13968439509059 a 
ShuffleNet Accuracy : 56.81818181818182 a 
Accuracy Comparison Graph | 2 
E 
MobileNetV2 Precision : 82.9059829059829 
MobileNetV2 Recall : $1.81818181818181 
pevormance Table MobileNetV2 FScore : 81.66666666666667 
MobileNetV2 Accuracy : $1.81818181818183 
Exit 3 
Total fine tuned features extracted from all algorit! D 
irs 
Fusion Model SVM Precision : 95.0 S 
Fusion Model SVM Recall : 96.1538461538461 = 
Fusion Model SVM FScore _ : 95.3684210526315: 
Fusion Model SVM Accuracy : 95.454545454545¢ 
Non Forged Forged 
Predicted class 
#lel>| +/Q\=| B) 
O Tose h 4 5 im m = EE lice a 
ype here to searc ls g T ra e] £ la 03-05-2022 


In above screen with Fine tune SVM fusion model we got 95% accuracy and in confusion matrix 
graph x-axis represents PREDICTED LABELS and y-axis represent TRUE labels and we can see 
both X and Y boxes contains more number of correctly prediction classes. In all algorithms we can 
see fine tune features with SVM has got high accuracy and now close confusion matrix graph and 
then click on ‘Run Baseline SIFT Model’ button to train SVM with SIFT existing features and get its 
accuracy 
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& Figure 1 - a 


Upload MICC-F220 Dataset | E:/venkat/2021/May22/Dataset I] Baseline SIFT SVM Confusion matrix 


Preprocess Dataset SqueezeNet Accuracy : 79.54545454545455 


ShuffleNet Precision : 62.74131274131274 
Generate & Load Fusion Model | ShuffleNet Recall: 56.81818181818181 
ShuffleNet FScore : 51.13968439509059 
ShuffleNet Accuracy : 56.81818181818182 
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Fine Tuned Features Map with SVM | 


Forged 


MobileNetV2 Precision : 82.9059829059829 
Run Baseline SIFT Model MobileNetV2 Recall: $1.81818181818181 
MobileNetV2 FScore : 81.66666666666667 


MobileNetV2 Accuracy : $1.81818181818183 


True class 


Accuracy Comparison Graph | 


Total fine tuned features extracted from all algorithmns : 57 


Peformance Talle Fusion Model SVM Precision : 95.0 


Fusion Model SVM Recall : 96.15384615384616 
Exit Fusion Model SVM FScore :95.36842105263159 
Fusion Model SVM Accuracy : 95.45454545454545 


Baseline SIFT SVM Precision : 67.94871794871796 

Baseline SIFT SVM Recall : 67.5 

Baseline SIFT SVM FScore : 67.57894736842105 

Baseline SIFT SVM Accuracy : 68.18181818181817 


Non Forged 


Non Forged Forged 
Predicted class 


alel?) H/Q\/=| m] 
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In above screen with existing SIFT SVM features we got 68% accuracy and in confusion matrix graph 
we can see existing SIFT predicted 6 and 8 instances incorrectly. So we can say existing SIFT features 
are not good in prediction and now close above graph and then click on ‘Accuracy Comparison 
Graph’ button to get below graph 
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In above graph x-axis represents algorithm names and y-axis represents accuracy and other metrics 
where each different colour bar represents different metrics like precision, recall etc. Now close above 
graph and then click on ‘Performance Table’ button to get result in below tabular format 
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In above screen we can see propose fusion model SVM with fine tune features has got 95% accuracy 
which is better than all other algorithms 


7.Conclusion 

Image forgery detection is a very challenging problem. In this era of technological advancement, we 
need to be able to distinguish between real and tampered images. In this study, we proposed a deep 
learning-based approach for image forgery detection. The proposed model is based on ResNet50v2 
architecture, which uses residual layers; thus, using this architecture increases the detection rate of 
tampered images. Using this approach also provides the benefit of transfer learning by using the pre- 
trained weights of the YOLO CNN model. The use of transfer learning enabled us to train our model 
more efficiently, as we initialized our proposed model by meaningful assigning weights. This reduced 
the training time and complexity of the model and makes the architecture more efficient. We 
evaluated our proposed architecture on benchmark datasets, CASIA_v1 and CASIA_v2. We also 
compared the performance of our system with and without the use of transfer learning. We obtained 
an accuracy of 99.30% with the CASIA_v2 dataset for the forgery detection problem. The results of 
the comparison with the existing methods show the superiority of the proposed system. The proposed 
system will help in the image manipulation detection domain and also paves the way for future 
research in detecting multiple types of image forgery manipulations. 
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